Linking Program Level Assessment to 
Course Level Assessment Activities 


Robert D. O’Keefe, Professor 

Driehaus College of Business 
Kellstadt Graduate School of Business 
De Paul University 
Chicago, Illinois 
Lawrence O. Hamer, Professor 

Driehaus College of Business 
Kellstadt Graduate School of Business 
De Paul University 
Chicago, Illinois 

Philip R. Kemp, Professor Emeritus 

Driehaus College of Business 
Kellstadt Graduate School of Business 
De Paul University 
Chicago, Illinois 

ABSTRACT 

Assessment, or better stated, the assurance of student learning, has become a central issue in both 
the internal and the external evaluations of degree programs offered by colleges and universities. 
The continual importance ofassurance oflearning activities within institutions of higher education 
has generated a growing need to create empirical assessment efforts that are both meaningful and 
relevant to whatever goals and objectives an institution of higher education plans to achieve. Most 
ofthe learning that the institutions seek to assure takes place in the context ofindividual courses. For 
that reason progress in making assessment measurements more impactful suggests an approach that 
links specific class assignments and activities to specifically desired learning outcomes. The present 
paper presents an Assessment Validation Model. The model points out the linkages between and 
among activities relevant to assessment. The paper also provides empirical data that both demon¬ 
strate the effectiveness of the AVM andfocus on validating the relationships between university lev¬ 
el assessment objectives and learning objectives designed to be achieved within an individual class. 


Introduction 

As a result of a variety of pressures and incentives, 
assessment activities have become a central issue 
for discussion, study and action on university 
campuses.(Black & Duhon 2003) Governmen¬ 
tal agencies, private foundations that provide 
funding for university programs, and accredit¬ 
ing agencies are increasingly requiring that in¬ 
stitutions of higher education provide credible 
evidence of the effectiveness of their programs in 
meeting the frequently lofty objectives and goals 
stated in their proposals for funding, in their 
bulletins and on their web sites ( AACSB 2007; 
Folger 1977; Martel 2007). Additional pressure 


is being applied by both students and parents 
faced with paying increasing tuition and fees 
(Burke 2005; Peterson 1999). Students are espe¬ 
cially concerned about whether the high level of 
debt they have assumed in order to pay their edu¬ 
cational expenses will, over time, yield a positive 
cost/benefit ratio (Business Week 2012, Porter 
2002 ). 

An additional source of pressure on universities 
to increase their assessment efforts is generated 
by the increase in the numbers of for-profit uni¬ 
versities in general and, more specifically, those 
institutions that stress a direct connection be¬ 
tween the programs they offer and specific fields 
of employment (Deming et. al 2011, Fain 2011). 
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In addition to these specialized programs com¬ 
munity colleges are being advised, or in some in¬ 
stances directed, to focus more on employment 
based programs of study (Goldstein et. al 2012, 
Page 2012). 

WHAT IS ASSESSMENT? 

Assessment is the act of measuring student learn¬ 
ing and the goal of an assessment system is to en¬ 
sure that curricula and programs are effectively 
meeting the needs of students and, especially in 
the case of a business school, the expectations of 
our constituent employers. Institutions of higher 
education have, over many years, created plans 
and implemented programs directed toward as¬ 
sessment or, as perhaps better stated, assurance of 
learning programs. These efforts are directed to¬ 
ward individual programs offered by individual 
colleges (see, for example, AACSB White Paper 
No. 3 (2007)). As such, assessment provides a 
mechanism for evaluating the impact of courses 
on students’ development, and allows for a data- 
driven process for the continuous improvement 
of curricula and programs. Palomba & Banta 
(1999), for example, define assessment as the sys¬ 
tematic collection, review and use of information 
about educational programs undertaken for the 
purpose of improving student learning and de¬ 
velopment. 

While there seems to be consensus regarding the 
importance of assessment, arriving at a compre¬ 
hensive and agreed upon operational definition 
of assessment seems to represent a very real chal¬ 
lenge. Take, for example, the multiplicity of ap¬ 
proaches to assessment suggested to business pro¬ 
grams by accreditors, colleagues, and assessment 
experts (AACSB 2003, 2007; Bennis & O’Toole 
2005; Martell & Calderon 2005; Pringle and 
Michel 2007; Martell 2007; Rubin & Martell 
2009; Suskie 2004; Walvoord 2004). This lack 
of consensus regarding assessment measurement 
seems to have led to operational definitions that 
are situation specific and often border on the id¬ 
iosyncratic (Eshenfelder, Bryan & Tee 2010). At 
one end of the range of assessment operational¬ 
izations is the long-held notion that assessment 
can be equated to grading. At the other end of 
the range, some educators believe that assessment 
requires a more abstract approach that measures 
learners’ empowerment (e.g., Novak 2002). Teft 
to their own interpretations about assessment 


activities, some faculty members seem to draw 
upon the classics and operationally define the 
concept as: “doing as I am doing” (Tallise 2003). 

It’s understandable why many faculty members 
hold to the idea that grades are the best mea¬ 
sure of assessment and serve as empirical proof 
of the assurance of learning. However, serious 
discussions of learning, for example, Hill (1970), 
among many others, indicate that grades are re¬ 
ally not a measure of “ learning” but a measure 
of” performance”. Performance is what we as 
faculty members test and evaluate and we do so 
precisely because we cannot empirically measure 
how much a student may have learned beyond 
the answers to the questions we have chosen to 
ask and the assignments we have chosen to re¬ 
quire (Berrett 2012). 

Assessment requires something beyond offering 
simple literary definitions of the concept and 
certainly beyond the simplified individualistic 
ostensive definitions based entirely on grades or 
subjective observations. These definitions are not 
completely wrong, they are just not complete and 
do not provide the information needed to deter¬ 
mine, to the satisfaction of all parties concerned, 
whether or not the data collected and analyzed 
represent demonstrable assurance of learning 
outcomes. Whatever the definition of the con¬ 
cept of assessment or assurance of learning, the 
evidence for and the validation of these concepts 
must be found in results yielded by empirical re¬ 
search. Assignments are the principle vehicle for 
delivering student feedback relevant to the assess¬ 
ment process. Student feedback is most effective 
when it can be explicitly connected to specifically 
defined learning goals and objectives (Araund 
& Wakefield 2006; Nichol & McFarlane 2006; 
Hattie & Timperly 2007; May & Tidwell 2007; 
Sampson & Betters-Reed 2008; Choi, Tong & 
Kelley 2010; Gikandi, Morrow & Davis 2011) 

The challenge, then, is to connect course-level ac¬ 
tivities with program level assessments. The As¬ 
sessment Validation Model (AVM) (see Figure 
1) outlines the appropriate steps for proceeding 
to assess an individual course or a program of 
courses. The process begins with a delineation of 
specific goals of the institution and proceeds to a 
choice of course objectives relevant to the institu¬ 
tional goals. Based upon the course objectives, the 
model requires that specific measurement meth¬ 
ods and the appropriate measurement outcomes 
that are interpreted as proof that these objectives 
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have been met be clearly stated. Beyond these op¬ 
erations the model provides for course-program 
connections as the model measures the relation¬ 
ship between specific assignments/activities and 
the learning goals of a course and/or program. If 
the several measurements are in agreement, this 
represents a validation of the course objectives 
and the institutional goals and so illustrates the 
relationship between the course objectives and 
institutional goals. 

ASSESSMENT: AN EXAMPLE STUDY 

In this paper the authors present a successful fac¬ 
ulty instituted approach to assessment. Our pur¬ 
pose was to demonstrate the relationship between 
University level goals and the learning objectives 
prescribed for sections of an individual course. 
While the course included many measurement 
methods (examinations, quizzes, various assign¬ 
ments), this particular assessment example will 
focus on a particular assignment called “Environ¬ 
mental Scan Reports. The purpose of the assign¬ 
ment was to reinforce student’s understanding of 
Marketing terminology and strategy by having 
them bridge the gap from concepts covered in 
the textbook to contemporary business strategies 
and tactics reported within the business or popu¬ 
lar press” (see O’Keefe, Kemp & Kelly 1996 and 
2006 for more information on this assignment). 
As described below, assessing the effectiveness of 
this assignment involved linking the assignment 
to learning objectives for the course and those for 
the university overall. 

Figure 2 presents the learning objectives for the 
course and those for the university. Assuring that 
these objectives have been satisfied requires proof 


derived from the outcomes of direct measure¬ 
ments of assignments such as the Environmental 
Scan. 

Methodology and Measurements 

The example study employed a six-item survey 
in which each question was related to course 
and university learning objectives (see Figure 2). 
The survey instructions informed the students 
that the Marketing Department was interested 
in assessing the degree to which the environ¬ 
mental scan assignments contributed to their 
understanding of basic marketing terminology, 
concepts, and strategy. The items were measured 
using five-point scale ranging from 1 (no contri¬ 
bution to my understanding) to 5 (significant 
contribution to my understanding). 

The survey was administered at the same time 
as the course teaching evaluations. Both of the 
forms were distributed and collected by a student 
assistant. The instructor was not present when 
the data were collected. The results of both sur¬ 
veys were analyzed independently and the results 
reported to each instructor. 

The course instructors had agreed that, in order 
to consider that a learning objective had been 
satisfied, at least 75% of the students surveyed 
had to report the course provided considerable or 
significant contribution to their student’s attain¬ 
ment of a given learning goal. In other words, a 
learning goal was considered to be satisfied if 75% 
of the responses fell into the top two response 
options. Anything less than 75% would indicate 
that more time should be spent on a particular 
learning goal. The data are reported in Table 1. 


Figure 1 

Assessment Validation Model 

Program Level 

Mission (Goals) 


Learning Objectives 


Course Level 

Learning Objectives 


Content 

Feedback for Validation and 
Calibration 

Activities/Assignments 

Assessment 
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Figure 2 

Course and University Learning Objectives 

Course Learning Objectives 

As measured by examinations, quizzes and a series of written environmental scan reports, stu¬ 
dents completing Marketing 301 are expected to: 

• Demonstrate recall and recognition of basic marketing terminology and the operational (mea¬ 
surement based) definitions of relevant terms. 

• Demonstrate familiarity with the basic elements of marketing strategy and the relationships 
between and among these elements. 

• Demonstrate an understanding of the controllable and the uncontrollable variables relevant to 
the success or failure of marketing programs, tactics and strategies. 

• Demonstrate an understanding of competitive advantage in a firm’s marketing programs and 
strategies. 

• Demonstrate the ability to bridge concepts discussed in the text and these same concepts 
appearing in articles within academic and practitioner publications and the popular business 
periodicals. 

• Demonstrate improvement in both oral and especially written communication. 

Relevant University-level Learning Objectives 

• Mastery of Content: A DePaul Graduate will establish mastery of a body of knowledge and 
skills in depth and breadth. 

• Articulate Communication: A DePaul graduate will be able to communicate articulately in 
both the spoken and written word, being able to read and listen critically in order to under¬ 
stand the conversation in progress, and to adjust diction and style to the anticipated audience, 
to the subject matter and to the purpose of the communication. The goal recognizes the neces¬ 
sity that a student’s ability to communicate, keep pace with the subtlety, precision and depth 
of the student’s knowledge, sensibilities and deliberative powers. 

• Critical and Creative Thinking: A DePaul graduate will be capable of thinking critically 
and creatively, integrating knowledge and ways of knowing, making reflective judgments, 
identifying significant ideas and their underlying assumptions, biases and presuppositions. 


Results 

Table 1 summarizes the results of the student 
survey data collected to determine whether the 
Environmental Scan assignment contributed to 
student learning. As the results reported in Table 
1 point out, in all but one instance, better that 
75% of the students surveyed reported that the 
assignment had a positive impact on meeting the 
learning objectives specified for the assignment. 
Also the researchers chose relevant university lev¬ 
el learning objectives and as can be seen in Table 
1, the learning objectives specified for the class 
are concordant with those university objectives. 


SUMMARY 

Assessment or Assurance of Learning is serious 
business. Assessment plans are important but in 
the end institutions of higher education will be 
evaluated by how well and how rationally these 
plans are implemented and the results that imple¬ 
mentation yields. Thus, institutions cannot sim¬ 
ply expect that the whole will somehow be great¬ 
er than the sum of its parts and that a mosaic of 
multiple, idiosyncratic assessment activities will 
somehow morph into a seamless portrait of as¬ 
sessment provides assurances that students have 
made appropriate gains while moving through a 
program of study. 
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Table 1 

Relationships Between Assessment Items and Learning Objectives for the 

Course and University Goals 

Class Learning 
Objective 

Assessment Items Distribution 
(n=137) 

University 

Learning 

Goal 

1 

2 

3 

4 

5 

Demonstrate recall and rec¬ 
ognition of basic market¬ 
ing terminology and the op¬ 
erational (measurement based) 
definitions of relevant terms. 

1.5% 

0.0% 

10.9% 

59.9% 

27.7% 

Mastery of Content 

A DePaul graduate will establish 
mastery of a body of knowledge and 
skills in depth and breadth. 

Demonstrate familiarity 

with the basic elements of 
marketing strategy and the 
relationships between and 
among these elements. 

0.0% 

1.5% 

16.1% 

56.2% 

26.3% 

Critical Thinking 

A DePaul graduate will be capable of 
thinking critically and creatively... 

Demonstrate an understand¬ 
ing of the controllable and 
the uncontrollable variables 

relevant to the success or fail¬ 
ure of marketing programs, 
tactics and strategies 

0.0% 

2.2% 

18.4% 

50.7% 

28.7% 

Critical Thinking 

A DePaul graduate will be capable of 
thinking critically and creatively... 

Demonstrate an understand¬ 
ing of the controllable and 
the uncontrollable variables 

relevant to the success or fail¬ 
ure of marketing programs, 
tactics and strategies. 

0.0% 

1.5% 

12.5% 

44.1% 

41.9% 

Mastery of Content 

A DePaul graduate will establish 
mastery of a body of knowledge and 
skills in depth and breadth. 

Demonstrate an understand¬ 
ing of competitive advantage 

in a firm’s marketing programs 
and strategies 

0.0% 

2.2% 

25.5% 

51.8% 

20.4% 

Critical Thinking 

A DePaul graduate will be capable of 
thinking critically and creatively... 

Demonstrate improvement in 
both oral and especially writ¬ 
ten communication. 

0.7% 

4.4% 

19.1% 

54.4% 

21.3% 

Articulate Communication 

A DePaul graduate will be able to 
communicate articulately in both 
the spoken and written word... 

Note: A grey box indicates that a learning objective was satisfied because at 75% or more of the 
respondents reported that the assignment led to a significant to considerable contribution toward 
a course learning objective. 


Colleges and universities are free to adopt and 
implement comprehensive assessment programs 
and to use the data derived from these programs 
to implement changes in various components of 
educational programs. Whatever the methods 
chosen, in the end the overarching goal of assess¬ 
ment programs is to encourage continuous im¬ 
provement and innovation which are recognized 
as critical for maintaining progress and growth 


in both the private and public sectors of society. 
Currently, most assessment activities measure 
students’ progress toward program-level learning 
goals. As most student learning takes place with¬ 
in individual courses, there exists a need to link 
the activities that happen within a course with 
the overall learning goals that are being assessed. 
The Assessment Validation Model (AVM) pro¬ 
vides a mechanism for linking course activities 
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and program-level learning objectives. Using 
this model, it was demonstrated that instructors 
can see the impact of particular assignments on 
students’ attainment of desired learning out¬ 
comes. Further, and perhaps most importantly, 
the information gained through the AVM can 
be used to highlight deficiencies in students’ de¬ 
velopment and links the deficiency to particular 
course activities. 

REFERENCES 

AACSB (2003). Eligibility standards for busi¬ 
ness accreditation. St. Louis Mo.: Association 
to Advance Collegiate Schools of Business. 

AACSB International Accreditation Coordinat¬ 
ing Committee, and AACSB International 
Accreditation Quality Committee (2007). 
AACSB White Paper No. 3: AACSB Assur¬ 
ance of Learning Standards:An Interpretation. 
St.Louis, MO: AACSB International. 

Aurand, T.W. & Wakefield, S.A. (2006) Meet¬ 
ing AACSB assessment requirementsthrough 
peer evaluations and rankings in a capstone 
marketing class. Marketing Education Re¬ 
view ,16 (Spring) 41-46. 

Black, H.T. & Duhon, D.L.(2003) Evaluating 
and improving student achievementin busi¬ 
ness programs: The effective use of standard¬ 
ized assessment tests. Journal of Education for 
Business 78 (November/D ecember) ,90-98. 

Burke, Joseph C. (2005). Achieving accountabil¬ 
ity in higher education: balancing public, aca¬ 
demic, and market demands, San Francisco: 
Jossey-Bass. 

Berrett, D. (2012). Harvard conference seeks 
to jolt university teaching. The Chronicle of 
Higher Education. February 5. 

Business Week (2012). Housing’s latest hurdle: 
Too much student debt. p. 49-50. 

Choi, B, Tong, P. & Kelley, C.A. (2010), An ex¬ 
ploratory study of student learning assessment 
in marketing programs. Journal for Advance¬ 
ment of Marketing Education, Vol. 16, Sum¬ 
mer 10-19. 

Deming, D, J., Goldin, C., & Katz L. F (2011). 
The for-profit postsecondary school sector: 
nimble critters or agile predators? Presented 
to Federal Reserve Bank of Atlanta, July 
2011. 


Eschenfelder, M., Bryan, L.D. & Lee, Tanya 
(2010). Assurance of learning: what do 
economics faculty know and what do they 
believe? Journal of Case Studies in Accredita¬ 
tion and Assessment, vol.l, Sept, (online) pp 
1-18. 

Fain, Paul (2011). Questions of qual¬ 
ity. Inside Higher Ed. accessed March 17, 
2012: http://www.insidehighered.com/ 

news//2011/12/07. 

Folger, J.K. (1997) Increasing the public ac- 
countabilityof higher education. New Di¬ 
rections for Institutional Research IV (4) San 
Francisco: Jossey-Bass. 

Gikandi, J.W., Morrow D., & Davis, N.E 
(2011), Online formative assessment in 
higher education: A review of the literature. 
Computers & Education, vol. 57(4), 2333- 
2351. 

Goldstein, H. A., Lowe, N. & Donegan, M. 
(2012), Transitioning to the new economy: 
Individual, regional and intermediation in¬ 
fluences on workforce retraining outcomes. 
Regional Studies, v ol. 46(1), 105-118. 

Hattie, J. and H. Timperly (2007) The power 
of feedback. Revieiv of Educational Research, 
vol 77(1), 81-112. 

Hill, W.F. (1970). Psychology: Principles and 
Problems. Philadelphia PA: J.B. Lippincott 
Company. 

Martel, K and Calderone, T. (2005), Assess¬ 
ment in business schools: What is it, where 
are we and where we need to go now. In K. 
Martell and T. Calderon (Eds), Assessment 
of Student Learning in Business Schools. 
(pp.1-26). Tallahassee, FL: Association for 
Institutional Research, vol.l, 1. 

Martell, K. (2007), Assessing student learn¬ 
ing: Are business schools making the grade. 
Journal of Education for Business, 82(March- 
April), 189-95. 

May, G. and Tidwell, M. (2007). Assurance of 
learning: Implementing a uniform assess¬ 
ment process across multiple sections of a 
managerial communication course. Proceed¬ 
ings of the Association for Business Commu¬ 
nication Annual Convention. http://business 
communication.org wo. content. 


166 


Spring 2013 (Volume 9 Issue 1) 



Linking Program Level Assessment to Course Level Assessment Activities 


Nicole D. & Macfarlane, D. (2006). Forma¬ 
tive assessment and self-regulated learning: 
A model and seven principles of good feed¬ 
back practice. Studies in Higher Education, 
vol. 31(2), 199-218. 

Novak, J. D. (2002), Meaningful learning: The 
essential factor for conceptual change in 
limited or inappropriate propositional hier¬ 
archies to empowerment of learners. Science 
Education, vol. 86(4), 548-571. 

O’Keefe, R.D., Kemp, P.R. & Kelly, J.S. (1996). 
Principles of marketing: using environmen¬ 
tal scan reports as a means of assessing stu¬ 
dent learning. In J.F. Hair, C.W. Lamb and 
S.S. Roach (Eds.), Great Ideas in Teaching 
Marketing (pp. 403-406).Cincinnati, OH: 
South-Western College Publishing. (2006). 
Principles of marketing: using environmen¬ 
tal scan reports as a means of assessing stu¬ 
dent learning. In J. Rupp and J. Bryant (Eds), 
,Handbook for Neiu Instructors: Getting 
Started ivith the Great Ideas, (pp.121-123). 
Mason, OH: Thompson-South-Western. 

Page, C.(2012). A GED for college? Chicago 
Tribune, Section 1 Sunday March 11 p. 21. 

Palomba,C.A. & BantaT.W. (1999). Assessment 
Essentials. San Francisco CA: Jossey-Bass 

Peterson, Marvin W. (1999), Designing Stu¬ 
dent Assessment to Strengthen Institutional 
Performance in Comprehensive Institutions, 
National Center for Postsecondary Im¬ 
provement, Stanford University, School of 
Education. 


Porter, Kathleen (2002). The value of a college 
degree. ERIC Clearinghouse on Higher Edu¬ 
cation (pp.403-406). accessed March 17, 
2012:http://www. ericdigests.org2003-3/ 
value.htm. 

Pringle, M. & Michel, M. (2007). Assessment 
practices in AACSB-accredited schools. Jour¬ 
nal of Education for Business ol.82,4 Mar.- 
April, 202-2111. 

Rubin, R. & Martel, K (2009). Assessment and 
accreditation in business schools. In S.J. Arm¬ 
strong & C.W. Fukomi (Eds.), The Sage Hand¬ 
book of Management Learning and Develop¬ 
ment (pp.364-384). Los Angeles CA: Sage 
Publishing. 

Sampson,S.D. & Betters-Reed, (2008) Assess¬ 
ment of learning and outcomes assessment: A 
case study of assessment of a marketing cur¬ 
riculum. Marketing Education Review, 18(3), 
25-36. 

Suskie, L. (2004). Assessing Student Learning: A 
Common Sense Guide. Bolton, MA: Anker 
Publishing. 

Tallise, R.E. (2003).Teaching Plato’s Euthyphro 
dialogically. Teaching Philosophy, 26, 2,163- 
175 

Walvoord, B. (2004). Assessment Pure Clear and 
Sitnple: A Practical Guidefor Institutions, De¬ 
partments and General Education. San Fran¬ 
cisco, CAJossey-Bass. 


Journal of Learning in Higher Education 


167 



